Automated bin system for accepting food items in robotic kitchen workspace

ABSTRACT

A robotic kitchen system for preparing food items in combination with at least one kitchen appliance such as a fryer comprises an automated bin assembly, a robotic arm, and a basket held by the robotic arm. The automated bin assembly comprises at least one automated bin for holding the food items. A camera or sensor array collects image data of the food items in the bin(s). A central processor is operable to compute and provide directions to the first robotic arm and automated bin assembly based on the image data and stored data to (a) move the robotic arm to the bin; (b) actuate the bin to drop the food items from the bin into the basket; (c) and to move the basket into the fryer all without human interaction. Related methods are also described.

CROSS-REFERENCE TO RELATED APPLICATIONS

This claims priority to U.S. application Ser. No. 17/727,363, filed Apr.22, 2022, entitled “AUTOMATED BIN SYSTEM FOR ACCEPTING FOOD ITEMS INROBOTIC KITCHEN WORKSPACE” and to provisional application No.63/182,912, filed May 1, 2021, entitled “AUTOMATED BIN SYSTEM FORACCEPTING FOOD ITEMS IN ROBOTIC KITCHEN WORKSPACE.”

BACKGROUND OF THE INVENTION 1. Field of the Invention

This invention relates to kitchen appliances and more particularly torobotic kitchen appliances capable of performing a wide range of foodpreparation steps in a restaurant environment.

2. Description of the Related Art

Providing a robust and effective apparatus or combination of apparatusesto prepare food for consumers is challenging because of the wide varietyof types of food, cooking techniques, kitchen appliances, kitchen tools,and utensils. Additionally, food preparation is often labor intensiveand subject to human error. Workers employed by these businesses requirecareful and sometimes excessive training to accurately and safelyprepare the food, thus increasing costs. It follows that businesses thatprepare and sell food typically have high labor costs and experiencelarge amounts of monetary and food loss as well as customerdissatisfaction due to human error.

Various commercial food preparation equipment addressed some of thesechallenges. The existing equipment, however, has several drawbacks.First, food preparation equipment is usually designed as a bespokesolution to perform a limited scope of work. Chaining together manydifferent pieces into a workflow is a complex and expensive process andresults in a system with highly limited capabilities yet a largefootprint. Second, such food preparation equipment typically requiresbatch preparation of food items. For example, clamshell grills aretypically designed to prepare multiple food items in batches and are notuseful for preparing items individually. Third, the increased mechanicaland/or electrical complexity inherent in such devices often leads toincreased failure rates versus conventional, non-automated versions ofsuch devices, resulting in higher downtimes. Such downtimes can beespecially costly for restaurants because restaurants do not typicallyhave back-up equipment onsite and consequently, they may not be able tocook a number of items on their menu which reduces average order size oreven drives away potential customers. Fourth, such food preparationequipment typically has a large footprint compared to conventionalversions of the equipment and for the variety of items they are capableof cooking. This larger size is a challenge for restaurant ownersbecause of the high cost of kitchen space. For example, the MasterMaticfryer is substantially larger than a similar model without the automaticconveyor. Fifth, the potential benefits of such food preparationequipment are often outweighed by their associated upfront costs. Forexample, automated frying equipment is significantly more expensive thanconventional frying equipment. Sixth, such food preparation equipmentstill requires extensive involvement of kitchen workers. Seventh, mostfood preparation equipment doesn't interact with outside data to achieveoptimal production scheduling and reduce food wasted. Because of theabove challenges, use of automation in restaurant and food serviceoperations is generally limited to spot solutions and does not takeadvantage of the availability of data to build accurate demand modelsand then use these models to automatically feed a production schedule.

Additionally, there are challenges associated with utensils for fryingfood items. Fry baskets are used to contain food during the cookingprocess in a fryer, enabling easy extraction from product. Analternative approach is termed “open-bay” cooking, where food is tossedin the fryer and then shoveled out. Baskets are traditionally made fromwire or sheet metal with silicone handles to provide thermal insulationfor a kitchen worker grabbing the basket. Despite the insulated handle,the worker is exposed to hot oil and hot equipment as well as the riskof slipping and falling arising from oil splatter in the vicinity of thefryer.

Accordingly, a robotic kitchen system and method that overcomes theabove-mentioned challenges is desirable.

SUMMARY OF THE INVENTION

A robotic kitchen system for preparing food items in combination with atleast one kitchen appliance in a commercial or restaurant kitchencomprises a robotic arm and end effector or tool, a basket held by theend effector, and an automated bin assembly in the vicinity of therobotic arm. The automated bin assembly includes a number of bins forholding the food items. A camera or sensor array is aimed towards thebins for collecting image data of the food items in the bin. A centralprocessor is operable to compute and provide directions to the firstrobotic arm and automated bin assembly to (a) move the robotic arm tothe bin, and (b) actuate the bin to drop the food items from the bininto the basket.

In embodiments, the central processor is further operable to compute andprovide directions to the robotic arm to move the basket into a fryerafter the basket receives the food items.

In embodiments, the robotic kitchen system further comprises a shieldand first window or opening in the shield through which at least aportion of the bin can protrude to receive the food items from outsidethe shield. In embodiments, the bin is operable to rotate from a firstposition in which a front portion of the bin protrudes through the firstwindow, and a second position in which the food items fall out a rearopening of the bin.

In embodiments, the central processor is further operable to compute andprovide directions to transfer the food items through a second windowafter the food items are cooked in the fryer.

In embodiments, the robotic kitchen system further comprises a ramp orchute extending through the second window, and arranged at a downwardangle such that the food items placed on the ramp slide down the rampout the second window.

In embodiments, the automated bin status comprises a plurality of bins,each of which is automatically rotatable.

In embodiments, the robotic kitchen system comprises a plurality offryers.

In embodiments, the robotic kitchen system further comprises ascheduling engine to determine a sequence of food preparation steps todetect and transfer the food items from each of the bins to the fryersfor cooking based on a plurality of inputs selected from the groupconsisting of camera data, customer orders, inventory, and recipeinformation.

In embodiments, the robotic kitchen system further comprises a frame, towhich the shield is attached, and a linear guide to which the roboticarm is movably coupled.

Automated Food Transfer Method to Robotic Kitchen Workspace

In embodiments, a method of robotically preparing food items in acommercial or restaurant kitchen having at least one kitchen appliancefor cooking a food item includes the following steps: detecting the fooditem placed in a bin; robotically manipulating a basket underneath thebin, wherein the robotically manipulating is performed with a roboticarm; and automatically moving the bin to transfer the food items to thebasket.

In embodiments, the method further comprises robotically manipulatingthe basket into the cooking appliance after the food item has beentransferred to the basket.

In embodiments, the method further comprises classifying, using aprogrammed processor and image data, the food item in the bin.

In embodiments, the method further comprises determining, using aprogrammed processor, a cook schedule for the food item based on type offood item.

In embodiments, the method further comprises removing the basketcontaining the food item from the cooking appliance after the food itemhas been cooked.

In embodiments, the method further comprises robotically dumping thecooked food item onto a target surface. In embodiments, the targetsurface is a ramp, and the ramp is shaped and arranged such that thefood item dumped thereon slides downward and falls into a food holdingarea.

In embodiments, the method further comprises providing a shield in frontof the robotic arm and fryer, and wherein the shield comprises a firstopening for a portion of the bin to protrude.

In embodiments, the step of automatically moving the bin is performed byrotating the bin using a motor or actuator removably coupled to the bin.

In embodiments, the method further comprises decoupling the bin from themotor and cleaning the bin.

In embodiments, the method further comprises controlling, using aprogramed processor, the robotic arm to robotically manipulate thebasket based on image data from a camera or sensor.

In embodiments, the method further comprises controlling, using aprogramed processor, the bin to transfer the food item to the basketbased on image data from a camera or sensor.

Automated Bin Station

An automated food bin station for transferring food items from a humanto robotic kitchen assistant comprises: a bin; a motor removably coupledto the bin and operable to move the bin between a first position forreceiving the food items and to a second position for dumping the fooditems; a sensor array for obtaining image data of the bin; a processorfor detecting and classifying food items in the bin, and for instructingthe motor to move the bin between the first position and the secondposition.

In embodiments, the automated bin station further comprises a rampextending through the second window, and arranged at a downward anglesuch that the food items placed on the ramp from within the enclosureslide down the ramp out the second window.

Optionally, a sorting bin for collecting the food from the ramp isarranged below the egress end of the ramp. In embodiments, a linear railand motor are adapted to move the sorting bin laterally and rotationallyand the central processor is further operable to compute and providedirections to move the sorting bin to at least one food holding tray,and to dump the food into the food holding tray.

In a work environment requiring a human to collaboratively work with arobot, a method for securing a robot workspace from a human workspace toprotect the human from injury arising from contact with the robotcomprises automatically presenting an object collection zone from therobot workspace through a window in a physical barrier into the humanworkspace, said physical barrier separating the human workspace from therobot workspace; and automatically manipulating the object collectionzone to transport an object placed in the object collection zone to therobot workspace, wherein the manipulating step is performed independentof the human and robot.

In embodiments, the method further comprises automatically detecting theobject when the object is placed in the object collection zone. Anexample of the object is a food item such as frozen fries or chickenfingers. Examples of the object collection zone include, withoutlimitation, a bin, chamber, container, and cart as well as spaceprovided on a conveyor belt assembly, rack, or guide system.

In embodiments, the method further comprises positioning a cooking toolunderneath a portion of the object collection zone, wherein thepositioning is performed by the robot.

In embodiments, the method further comprises automatically moving theobject collection zone to transfer the object to the cooking tool. Anexample of a cooking tool is a fry basket.

In embodiments, the method further comprises robotically moving thecooking tool to dump the object onto a pathway, wherein the pathway isconfigured to transport the object from inside the robot workspace tooutside the robot workspace. In embodiments, the pathway is a chute, andthe chute is arranged with a downward sloping angle to transport theobject based on gravity.

In embodiments, the step of automatically manipulating is performed byan automated bin assembly comprising a plurality of computer-controlledbins, each of which is adapted to rotate or tilt from a firstorientation to receive the object from the human in the human workspace,to a second orientation to position the object inside the robotworkspace.

An automated sorting station as described herein.

An automated bin station as described herein.

A method for transferring food items from a bin to a basket as describedherein.

A robotic kitchen system operable to transfer food items from a bin to abasket as described herein.

The description, objects and advantages of embodiments of the presentinvention will become apparent from the detailed description to follow,together with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a modular robotic kitchen system comprising a plurality ofmodular carts in accordance with an embodiment of the invention;

FIGS. 2-4 show various views of another modular robotic kitchen systemarranged in a commercial kitchen environment in accordance with anembodiment of the invention;

FIG. 5 is an illustration of a flexible system architecture of a robotickitchen system, and optionally a modular robotic kitchen, in accordancewith embodiments of the invention;

FIG. 6 is a flow diagram for a robotic temperature testing system inaccordance with an embodiment of the invention;

FIGS. 7A-7C show an example configuration of camera positions andorientations for a robotic temperature testing system in accordance withan embodiment of the invention;

FIG. 8 shows a vibrating rack mechanism which allows a bin to beagitated easily in accordance with an embodiment of the presentinvention;

FIG. 9 shows a bin resting securely in the vibrating rack shown in FIG.8 in accordance with an embodiment of the present invention;

FIG. 10A shows a side view of the Temperature Testing Tool in theextended position in accordance with an embodiment of the presentinvention;

FIG. 10B shows a side view of the temperature testing tool of FIG. 10Ain the retracted position;

FIG. 11 shows a design of a robot-friendly freezer package withoverlapping seams and gripper system in accordance with an embodiment ofthe present invention;

FIG. 12 shows a design of a robot-friendly freezer package opened bygripper system in accordance with an embodiment of the presentinvention;

FIG. 13 is a flow chart illustrating an operational procedure forpacking a food container in accordance with an embodiment of theinvention;

FIG. 14 shows an overhead view of a robotic food packing system inaccordance with an embodiment of the invention where the unsorted foodboxes represent locations where bins of unsorted food are placed and thepacking area is a work area where packing containers can be placed whilethey are being packed;

FIGS. 15A and 15B are a side and front views, respectively, of a roboticfood packing system in accordance with an embodiment of the invention;

FIG. 16 shows a robotic arm with an opposable gripper capable of pickingup a variety of food items in accordance with an embodiment of theinvention;

FIG. 17 shows a measuring tool that is capable of using a variety ofmeasuring tools for liquid and powders in accordance with an embodimentof the invention;

FIG. 18A shows a basket design with computer vision (CV) marker mountplate and diamond in accordance with an embodiment of the invention;

FIG. 18B shows a basket design with T-shaped feature on top of basket toadd more vertices to object with sharp edges in accordance with anembodiment of the invention;

FIG. 19 shows a basket design with implement for easy dumping of basketwithout lifting full weight of basket in accordance with an embodimentof the invention;

FIG. 20 shows a mobile robot transporting food between modular units inaccordance with an embodiment of the invention;

FIG. 21 shows a hot case, and sensors mounted thereon to observe thecontents of the hot case to estimate the available quantity of foodremaining in accordance with an embodiment of the invention;

FIG. 22 is a flow chart for a method to control the actions of variousrobotic kitchen assistants in a robotic kitchen in accordance with anembodiment of the invention;

FIG. 23 is a block diagram of a conveyor system for routing food betweentwo or more robotic kitchen assistants in accordance with an embodimentof the invention;

FIGS. 24A, 24B show a modular robotic kitchen unit comprising ahuman-robot drawer interface in closed configuration and openconfiguration, respectively, in accordance with an embodiment of theinvention;

FIG. 25 shows a robotic linear guide rail system mounted on a frameabove the cooking area in accordance with an embodiment of theinvention;

FIGS. 26A, 26B show, respectively, a front and left side view of arobotic kitchen system comprising a plurality of automated bins inaccordance with an embodiment of the invention;

FIG. 27 shows a plurality of automated bins arranged to receive food inaccordance with an embodiment of the invention;

FIG. 28 shows the plurality of automated bins of FIG. 27 except one binis shown in an actuated position for dumping food into a basket inaccordance with an embodiment of the invention;

FIG. 29 is an enlarged perspective view of an automated bin inaccordance with an embodiment of the invention;

FIG. 30 shows an enlarged perspective rear view of an automated binstation in accordance with an embodiment of the invention;

FIG. 31 is a flow chart for a method to receive food, cook and transferhot food to a holding area in accordance with an embodiment of theinvention;

FIGS. 32A, 32B show an upper front perspective view and a front view,respectively, of a robotic kitchen arm manipulating the fry basket tohot food ramp in accordance with an embodiment of the invention;

FIG. 33 shows an automated sorting system in accordance with anembodiment of the invention; and

FIGS. 34a-34c sequentially illustrate an automated sorting systempositioning a sorting bin under a chute to catch food items, laterallytransporting the sorting bin over a food holding tray, and rotating thesorting bin to dump the food items into the food holding tray.

DETAILED DESCRIPTION OF THE INVENTION

Before the present invention is described in detail, it is to beunderstood that this invention is not limited to particular variationsset forth herein as various changes or modifications may be made to theinvention described and equivalents may be substituted without departingfrom the spirit and scope of the invention. As will be apparent to thoseof skill in the art upon reading this disclosure, each of the individualembodiments described and illustrated herein has discrete components andfeatures which may be readily separated from or combined with thefeatures of any of the other several embodiments without departing fromthe scope or spirit of the present invention. In addition, manymodifications may be made to adapt a particular situation, material,composition of matter, process, process act(s) or step(s) to theobjective(s), spirit or scope of the present invention.

Methods recited herein may be carried out in any order of the recitedevents which is logically possible, as well as the recited order ofevents. Furthermore, where a range of values is provided, it isunderstood that every intervening value, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range is encompassed within the invention. Also, it iscontemplated that any optional feature of the inventive variationsdescribed may be set forth and claimed independently, or in combinationwith any one or more of the features described herein.

All existing subject matter mentioned herein (e.g., publications,patents, patent applications and hardware) is incorporated by referenceherein in its entirety except insofar as the subject matter may conflictwith that of the present invention (in which case what is present hereinshall prevail).

Described herein is a modular robotic kitchen system.

OVERVIEW

FIG. 1 shows a cooking system 10 in accordance with an embodiment of thepresent invention. The cooking system 10 shows a plurality of modularunits including an unpacking or ingredient unit 20, robotic kitchenassistant unit 30, robotic extension unit 40, packing unit 50, andwarming or pick-up unit 60, each of which is discussed herein. Each ofthe modular units are shown including a shielded workspace, a cart, andwheels to facilitate locating and relocating each of the carts.

Also shown in FIG. 1 is an optional transport robot 70 to assist movingthe modular units and to transport food and supplies between the modularunits. See also mobile transport 910 in FIG. 20 positioning a modularcart 920 adjacent a main robotic arm module 930. An exemplary sled foruse in the subject invention is described in U.S. patent applicationSer. No. 16/281,088, filed Feb. 20, 2019, and entitled “ROBOTICSLED-ENHANCED FOOD PREPARATION SYSTEM AND RELATED METHODS.”

RKA Module/Unit

The robotic kitchen assistant (RKA) module 30 is shown including ashielded workspace, counter-top or bin area, a robotic arm having aplurality of degrees of freedom (preferably, 6 DOF), at least one sensoror camera, and a computer operable to control the motion of the roboticarm to carry out food preparation steps as discussed further herein.Examples of an RKA and robotic arm suitable are described in U.S. patentapplication Ser. No. 16/490,534, filed Aug. 31, 2019, entitled “ROBOTICKITCHEN ASSISTANT FOR PREPARING FOOD ITEMS IN A COMMERCIAL KITCHEN ANDRELATED METHODS” and US Patent Publication No. 20180345485, filed Aug.10, 2018, and entitled “MULTI-SENSOR ARRAY INCLUDING AN IR CAMERA ASPART OF AN AUTOMATED KITCHEN ASSISTANT SYSTEM FOR RECOGNIZING ANDPREPARING FOOD AND RELATED METHODS.”

Unpacking/Ingredient Module

The unpacking or ingredient cart 20 is shown including a shieldedworkspace, and four separate areas for holding ingredients or bins ofingredients. As discussed further herein, in embodiments, the ingredientcart 20 can hold multiple food items (up to 10), is robot friendly;includes face protection, and a lid or cover to close. Optionally, oneor more of the separate areas are refrigerated. Additionally, inembodiments, discussed further herein, the system employs raw foodpackaging facilitating robot actions. By ‘food items’, it is meant toinclude a wide variety of types of food items whether cooked oruncooked.

Cooking Appliances

The modular robotic kitchen system can operate with a wide range ofcooking appliances (e.g., fryer 80, grill 90) as shown in FIGS. 2-4, andas discussed further herein. The robotic arm(s) are operable to movefood items to and from the applications to cook.

Preferably, in embodiments, temperature of the food items being cookedis monitored. The temperature can be input to scheduler engine,described further herein. Additionally, in embodiments, the temperaturein the appliances (e.g., fryer oil, oven temperature, grill surface,etc.) can be monitored and automatically controlled, discussed furtherherein.

Additionally, in embodiments, the modular robotic kitchen system caninclude various utensils to facilitate transferring from one station orcart to another. In a particular embodiment, a fry basket is operablewith the fryer and enables convenient and safe transfer of the frieditems to another unit or workspace, discussed further herein.

Assembly & Packing Module

FIGS. 1-4 show an assembly and packing module 50. The packing moduleunit 50 is shown having a shielded workspace, counter and/or bins forsupporting plates, dishes, bowls, or packing on which to serve or shipthe completed entree. Packing may be carried out in various manners,discussed further herein.

Warming Module

FIGS. 1-4 show a warming module 60 for holding completed entrees.

The warming module shown in FIG. 1 includes an enclosed space,temperature controlled, shelves optionally automatically movable toreceive and present a completed entrée, and includes sensors to monitortemperature and confirm contents and inventor therein. Completed entreesmay also be transported to the hot or cold cases 92, 94.

Extension Module

FIGS. 1-4 show an RKA extension module 40 to enhance, amongst otherthings, the reach, speed, and capability of the kitchen system. The RKAextension module is shown having a shielded workspace, small counter-toprelative to the RKA cart 30, a robotic arm having a plurality of degreesof freedom (preferably, 6 DOF), at least one sensor or camera, andoptionally a secondary computer operable to control the motion of therobotic arm. Optionally, a main computer controls the motion of both themain RKA cart and the extension module. Examples of an RKA and roboticarm suitable for the extension cart 40 are described in U.S. patentapplication Ser. No. 16/490,534, filed Aug. 31, 2019, entitled “ROBOTICKITCHEN ASSISTANT FOR PREPARING FOOD ITEMS IN A COMMERCIAL KITCHEN ANDRELATED METHODS” and US Patent Publication No. 20180345485, filed Aug.10, 2018, and entitled “MULTI-SENSOR ARRAY INCLUDING AN IR CAMERA ASPART OF AN AUTOMATED KITCHEN ASSISTANT SYSTEM FOR RECOGNIZING ANDPREPARING FOOD AND RELATED METHODS.”

System Architecture

FIG. 5 is block diagram illustrating the system architecture 100 of arobotic kitchen system in accordance with an embodiment of theinvention. With reference to FIG. 5, a core platform 110 includeshardware 120 and software 130.

Examples for use with embodiments of the inventions of hardware andsoftware include, without limitation, central computer, servers,processors, memory and storage, commination interface, sensors, cameras,input devices such as keyboards or touchscreen displays, display. Theprocessor is programmed or operable to execute various applicationsdescribed herein as well as enable modules or engines for determininglocation and identification of food items, doneness, scheduling ofsteps, demand of food items, and inventory. Examples of foodidentification and location, scheduling, and demand modules as descriedin U.S. patent application Ser. No. 16/490,534, filed Aug. 31, 2019,entitled “ROBOTIC KITCHEN ASSISTANT FOR PREPARING FOOD ITEMS IN ACOMMERCIAL KITCHEN AND RELATED METHODS”, U.S. patent application Ser.No. 16/490,775, filed Sep. 3, 2019, entitled “AUGMENTED REALITY-ENHANCEDFOOD PREPARATION SYSTEM AND RELATED METHODS”, and US Patent PublicationNo. 20180345485, filed Aug. 10, 2018, and entitled “MULTI-SENSOR ARRAYINCLUDING AN IR CAMERA AS PART OF AN AUTOMATED KITCHEN ASSISTANT SYSTEMFOR RECOGNIZING AND PREPARING FOOD AND RELATED METHODS”, each of whichis incorporated by reference in its entirety for all purposes.

Core platform additionally shows skills 140 that are enabled by thehardware and software. Collectively, the core platform is highlyflexible and adaptable to perform a wide range of cooking applications150 which may include specific cooking workflows 160 and use of specificcooking equipment 170 such as a burger workflow and use of a griddle,respectively. The core platform 110, as described further herein, isreadily adaptable to run a specific cooking workflow and use theprovided equipment without needing to be reworked or rewired.

In embodiments, a new cooking workflow software is downloaded to thecentral computer for execution. Optionally, trained models may bedownloaded to the central computer or the system trains itself based onmachine learning algorithms.

FIG. 5 also shows a support layer 170 including monitoring 172,continuous learning 174 and performance analytics 176.

The monitoring system 172 is operable to continuously track the statusof the system and flags anomalous behavior to be corrected by local orremote staff.

The continuous learning system 174 is operable to utilize these flaggedissues to retrain the neural networks in order to improve theperformance of the autonomous system for food classification.

A performance analytic system 176 is operable to aggregate at regularintervals to improve store management and give guidance on where tofocus efforts. The analytics serve to determine the difference betweenthe amount of food cooked compared to the amount of food ordered, toproduce food safety and quality reports, and to report on the status ofthe machine and when the next maintenance cycle is due.

Unpacking & Raw Food Packaging

In embodiments, a method for packing, transporting, and unpacking rawfood for preparation in kitchens includes providing custom containersdesigned for ergonomic access by humans and manipulation by mechanizedsystems.

Preferably, the raw packing system is in a centrally locateddistribution warehouse and is operable to quickly unpack and repack themodular carts. Additionally, the contents in each cart are trackedthroughout the time the contents are in the cart using an automatedtracking system.

The raw packing system can include various hardware such as a batteryand power management system, a charging interface to supply power to thebattery and power management system in the cart, and a wired and/orwireless communication system to maintain in-transit tracking of thecart and also to communicate with the robotic kitchen assistant modularunit, described herein.

In embodiments, an access control system is provided with the cart andoperable to obtain a return merchandise authorization (RMA) and to allowthe contents in the cart to be returned safely back to the distributionwarehouse and repacked for a different store, without riskingstore-to-store contamination.

The packing and unpacking system can optionally log environmental dataof the cart at all times.

The packing and unpacking system may include an environmental controlsystem to control the temperature and other environmental conditionswithin he warehouse or kitchen. For example, in embodiments, theenvironmental control system comprises a compressor-based bidirectionalheat pump, and optionally the heat pump may be a solid-state heat pumpusing, e.g., Peltier junctions.

In embodiments, the environmental control system comprises a passivethermal reservoir utilizing ice or other similar latent heat of phasechange and heavy insulation. In embodiments, a combination of the abovethermal control systems are used in combination.

In embodiments, the raw food is packaged in a thermal insulativecontainer. In particular embodiments, raw food is packaged in pillowpacks that are hermetically sealed via plastic welding. The pillow packis opened via a blade or a perforation in the packaging material, andthe contents can then be dumped into cooking container, e.g., fryerbasket or pot. The packaging material is then discarded.

In embodiments, the pillow pack container implementation can be grabbedwith a suction cup.

In embodiments, the pillow pack container can be grabbed with a moldedgripping feature designed for a custom end effector to enhancemanipulability of pillow pack.

In embodiments, small reusable rigid containers are used to contain foodproduct. Preferably, in embodiments, a freezer safe package can beopened without the use of a knife by pulling apart the bag.

With reference to FIGS. 11-12, in embodiments, a freezer-safe package510, 510′ that can be easily opened by a mechanized system 500, 500′ byfolding the packaging in half. FIG. 11 shows the design of arobot-friendly freezer package 510 with overlapping seams 512 andgripper system 500 in accordance with an embodiment of the presentinvention. FIG. 12 shows the design of the robot-friendly freezerpackage 510′ opened by gripper system 500′ in accordance with anembodiment of the present invention.

In embodiments, the freezer safe package encodes information about theproduct.

In embodiments, a freezer safe package is adapted to dissolve in hot oilto release contents into the oil to cook. Exemplary materials for thefreezer safe bag include rice paper, starch, etc.

Temperature Testing

In embodiments, a robotic assisted method for determining thetemperature of food being cooked (e.g., batch of fried foods) comprisessingulating the pieces of cooked food from a batch, ranking the piecesaccording to size, and testing the internal temperature of the largestpieces to guarantee food safety requirements.

FIG. 6 is a flowchart of a method 200 in accordance with an embodimentof the invention to test temperature of food items during cooking.

Step 210 states to insert a bin of the food items in a vibrating rack.With reference to FIGS. 7A-7C, a cart including a bin 330 is shown. Thebin 330 may be placed in the rack using the robotic arm 302.

An example of a vibrating rack mechanism which allows a bin to beagitated easily is shown in FIGS. 8-9. Particularly, FIG. 9 shows theBin 330 positioned securely in the vibrating rack 360. The vibratingrack 360 is designed to hold a bin and agitate it. This causesingredients to spread out on the bottom of the bin, thereby singulatingindividual pieces. Preferably the system has a vibrating rack which canhold a standard size bin. A standard size bin ranges from 3 inches by 6inches to 12 inches by 20 inches.

The bin shown in FIG. 8 also includes a camera 340, serving tosee/detect the contents within the bin, discussed further herein.

Step 220 states to vibrate for 30 seconds, or until the food items areseparated from one another. Steps 210 and 220 collectively serve tosingulate the food items.

Step 230 states to capture images of the food items from a plurality ofcameras. FIGS. 7A-7C show an example configuration of camera positionsand orientations 310. Additionally, in embodiments, the robot arm has acamera on its wrist, or another portion of the robot arm. The data fromthis robot arm-carried camera can be combined with the data from othercameras to improve accuracy by filling in sensing gaps.

Examples of the sensors include, without limitation, cameras (opticaland/or IR) and Time-of-Flight sensors (depth measurements). The array ofcameras 310 serves to provide enough information to estimate volume fromreconstructed 3D models, discussed further herein. Additionally, the bincan be made of highly transparent material to allow vision from thebottom.

Step 240 states to reconstruct the 3D model of the food items. Therobotic temperature testing system performs this analysis using thearray of cameras and performing a technique called stereo reconstructionas described in, for example, Ju Yong Chang, Haesol Park, In Kyu Park,Kyoung Mu Lee, Sang Uk Lee, GPU-friendly multi-view stereoreconstruction using surfel representation and graph cuts, ComputerVision and Image Understanding, Volume 115, Issue 5, 2011, Pages620-634. In embodiments, the images from the plurality of cameras arefused together using Stereo Reconstruction to obtain a 3D scan of thebins and the objects therein.

In embodiments, segmentation is achieved using a neural network as inKaiming He, Georgia Gkioxari, Piotr Dollar, and Ross B. Girshick, MaskR-CNN, arXiv, 2017. Analyzing the segmentation can determine whether thefood items have been fully singulated as well as provide a list ofobjects of interest.

Step 250 states to identify the largest pieces. In embodiments, for eachpiece of food, the system performs a volumetric analysis. Particularly,the segmented pieces are analyzed to see which are the largest withselect geometric calculations to find the largest part of a piece offood. The pieces can be ranked according thickness of the thickest part.One or more of the thickest pieces are then selected for temperaturetesting, discussed below.

Step 260 states to compute the optimal angle of approach and penetrationdepth for the temperature probe discussed further herein. This approachand penetration step is calculated based on the information, size andorientation determined from the above steps. In embodiments, it isdesirable to aim towards the center of mass of the food item, and of thelargest food item.

In order to test a given piece of food properly, an appropriate angle ofapproach and penetration depth must be selected. For homogeneous items(such as a piece of boneless chicken breast), it is sufficient to locatethe largest cross-sectional area and penetrate orthogonally to thesurface and up to the middle of the food item.

For items that do not have reasonably homogeneous heat capacity, such asa bone-in chicken breast, it is not sufficient to simple insert into thelargest cross section area. For the example of bone-in chicken breast,it's important to these the thickest piece but avoid the bone since itheats much faster than the surrounding tissue. Therefore, a model isnecessary to infer optimal angle of approach and penetration depth.

Learning a model for angle of approach and penetration depth can beaccomplished either through heuristic approaches or using machinelearning. With either approach, the goal is to build a model to estimatethe pose and parameters of a food item. Using this model, someembodiments use heuristics to specify how to approach and penetrate.

In embodiments, a heuristic model is sometimes used, such as located thelargest cross-sectional area and penetrating orthogonal to that. Thistype of method can work well on a variety of food items. But some fooditems require more complicated techniques.

Other embodiments use learning by demonstration to build a model forangle of approach and penetration. In embodiments, a thermal probe thatpublishes its pose in space is used by a human trainer. The humantrainer goes through the motions and the pose of the thermal probe istracked over time as the human trainer tests many pieces of a type offood item. Using these data, a model can be trained that will allowcomputation of the optimal angle of approach and penetration depth.

These models for computing optimal angle of approach and penetrationdepth are generated using shared data via the Internet. This allowsmultiple robotic temperature testing systems to learn more quickly.

Step 270 states to move and insert the probe. In embodiments,temperature testing is performed with a temperature testing tool 400 andprobe 420 attached to the end of a robot arm 302. This robot arm 302 canhave 4, 5, 6, 7, or a higher number of degrees of freedom. The robot armcan also take other configurations including but not limited to that ofa SCARA arm or a delta arm.

In the embodiment shown in FIGS. 10A, 10B, a thermal probe 420 on thetemperature testing tool 400 can be retracted and extended.

FIG. 10A a side view of the temperature testing tool 400 in the extendedposition in accordance with an embodiment of the present invention isshown. It can be seen that the probe tip 410 extends beyond the flange420 to allow for insertion into food items.

FIG. 10B shows the temperature testing tool of FIG. 10A in the retractedposition. It can be seen that the probe tip retracts beyond the flange430 to create separation from food items. The flange 430 makes contactwith the food to allow the probe to be retracted easily. The flangefacilitates separating the tested food item from the probe.

The extension may be performed by various mechanisms such as, e.g., aloaded spring 440, a pneumatic actuator, or an electromagnetic actuatorsuch as a motor. Retraction can be accomplished with a pneumaticactuator or an electromagnetic actuator. Preferably, the extensionaction is performed using a sufficiently fast actuator to cause theextending probe to quickly penetrate food. By moving quickly enough, theprobe is able to avoid static friction altogether and operate withkinetic friction which allows for less friction overall. This mitigatesundesired motion of the food item being tested that would otherwiseoccur during insertion of the thermal probe

The probe may be made of various materials including, e.g., stainlesssteel or another food-safe material with appropriate thermal propertiesthat can be inserted into a variety of cooked foods including but notlimited to bone-in chicken, chicken tenders, and chicken nuggets,hen/turkey parts, boneless chicken/turkey pieces, steaks, hamburgers,fillets, tenders, cutlets, potato wedges, etc.

In embodiments, the thermal probe has axial force sensing. This forcesensing provides feedback if the probe makes contact with a bone in apiece of meat or if a probe makes contact with any other impenetrablecomponents in a piece of food. In spring-loaded embodiments of thethermal probe, the force can be sensed by measuring the displacement ofthe probe from full extensions and applying Hooke's Law. Inelectromagnetic embodiments, current and dynamics can be measured andcompared against a model of expected current.

Step 280 states to record the temperature reading.

Step 290 states to inform user testing is complete.

Additionally, in embodiments of the invention, a sanitation is performedwhen a piece of food is measured to be below the food-safe temperaturethreshold. The sanitation step may be performed variously. In oneembodiment, the probe is sanitized with an attached sanitation bath. Thesanitation bath uses approved chemicals to sanitize the thermal probeand flange.

Equipment Temperature Integration with Robotic System

The modular kitchen systems described herein may also monitor andcontrol temperature of the appliances (e.g., a fryer or oven) duringoperation.

In one embodiment, a method for controlling kitchen equipmenttemperature includes selecting the optimal input at present time whileoptimizing for a time horizon based off of future thermal loadprediction and oil life preservation goals.

Oil life preservation may be performed, for example, by dropping thetemperature of kitchen equipment such as a fryer to extend the lifetimeof consumables such as fryer oil during periods when equipment is not inuse, as determined by a kitchen production forecasting system.

Additionally, the present invention includes preemptively changingthermal input into the kitchen equipment before a thermal load isapplied. For example, fryer gas burner can be turned on 20 secondsbefore a basket of frozen fries is dropped into fryer.

Preferably, control of the equipment is automated. In embodiments, acontroller utilizes a camera or sensors to track workers in the kitchento predict when food will be added to system. The controller raises orlowers the temperature of the appliance automatically based on thelocation and movement of the workers.

In embodiments, the controller is connected to a production forecastingsystem based on various inputs. Examples of input to the productionforecasting system include, without limitation: prior demand,point-of-sale data, and product stock levels.

In embodiments, the controller is connected to a robotic kitchenassistant which relays its cooking cadence over to controller forpredictive temperature control.

In embodiments, the computer monitors the health of the kitchenequipment by observing effect of heat input on temperature readouts whenequipment has no thermal load.

In embodiments, the robot is operable to skim contents out of the fryerto preserve the lifetime of the equipment and the oil.

In embodiments, the system determines optimum lifetime of the oil, andwhen the oil needs to be changed based on tracking the throughput offood cooked in the fryer.

Robotic Food Packing System

FIG. 13 shows an operational procedure 600 for packing a food container.For facilitating understanding of the invention, the procedure set forthin FIG. 13 will be discussed with reference to the apparatus shown inFIGS. 14-15B.

Step 610 states to insert bin of unsorted food. With reference to FIGS.14-15B, one or more bins of unsorted food 720 are placed in the unsortedfood area 702 within the workspace of the modular packing cart 700. Theunsorted work areas 702, and bines of unsorted food 720 are within arm'sreach of the robotic arm 710. Optionally, the system interfaces witheither a human kitchen worker or another robotic kitchen assistant toplace the bins or unsorted food in the unsorted food area 720.

Step 620 states to place at least one packing container 730 insidepacking area 704. In embodiments, one bin sits in the work area to beused for packing. Another bin sits in the work area and contains packingcontainers. However, the number of the bins and areas may vary.

Step 630 states to capture images of the unsorted food. The cameras orsensors 760, described herein, can be arranged above the worksurface andfood items or elsewhere to aim at and obtain images from multiple anglesof the unsorted food. With reference to FIG. 15B, the locations of theoverhead cameras 760 in the camera array are placed above theworksurface. Placing cameras in these locations allows the system toperceive the contents of Bins with high accuracy.

Indeed, in order to properly portion and plate or pack a container, aRobotic Food Packing System can see in 3D the objects inside a bin ofunsorted food. This 3D imaging data can then be used to drive decisions,discussed herein, on how and what to pick out of the bin of unsortedfood. High fidelity 3D vision in a Robotic Food Packing System isachieved with an array of optical cameras mounted above the WorkingSurface of the Cart. These cameras point at the various work areas as inFIG. 14.

Step 640 states to reconstruct the 3D model. Preferably, as discussedabove, stereo reconstruction is employed for this step.

Step 650 states to segment and classify the food items. This step may becarried out as described above.

Step 660 states to compute an optimal grasp approach for a piece offood. This step may be determined and carried out based on parameters ofthe end effector tool 770 and the robot arm 710, and the output fromstep 650.

Step 670 states to execute grasp.

Step 680 states to place food pieces in appropriate configuration inpacking container 730. This step is executed by the robotic arm, andbased on order information. In embodiments, pick and place is achievedusing computer vision. Images are captured by video cameras andprocessed by convolutional neural networks. Such a network involves aseries of convolutional layers and max pool layers as well as otherlayers. These neural networks can be trained to infer the optimal angleof approach and determine the path necessary to successfully pick up anobject.

Step 690 states to remove bin of packed containers. Optionally, similarto step 610, the system is operable to interface with either a humankitchen worker or another robotic kitchen assistant to remove the bin ofpacked food containers from the packing area 730.

As mentioned herein, the workspace of the modular cart may be shieldedto protect workers. In embodiments, and with reference to FIG. 15A,transparent windows 780 can be incorporated into the cart, rising fromthe waist-level work surface to prevent kitchen workers from interactingunsafely with the system.

In embodiments, various types of gripping, grasping, wedging, squeezing,clamping, scooping, ladling, skewering, and suctioning tools are used topick up one or more pieces of food. With reference to FIG. 16, forexample, the robotic arm 710 may be provided with an opposable gripper740 capable of picking up a variety of food items. Alternatively, withreference to FIG. 17, the robotic arm may be provided with a measuringtool 750 that is capable of using a variety of measuring tools forliquid and powders.

In embodiments, sorting and packing is performed with a gripper toolattached to the end of a robot arm. The robot arm can have 4, 5, 6, 7,or more degrees of freedom. Additionally, the robot can have otherconfigurations including but not limited to, a SCARA or delta-typerobot.

In embodiments, the robot arm may have a camera on the wrist. The datafrom this camera can be combined with the data from other cameras toimprove the accuracy of pick and place behaviors. In embodiments, thewrist imaging sensor may be RGB, IR, or depth, or some combination ofthese sensors.

In embodiments, a convolutional neural network is sometimes used toidentify packing containers, either in a stack or set out in preparationfor packing.

In embodiments, the decision on what and how to pack is driven byexternal data coming in via sensors and the Internet. Packing contentsare determined by recipes.

In embodiments, learning by demonstration is sometimes used to build amodel for picking up food items. A human expert goes through the motionsof picking up many examples of a food item or various food items. Thesedata can be used to train a model to pick up various food objects.

In embodiments, reinforcement learning (trial and error) is used. Inthis process, the system makes repeated attempts at picking up foodobjects. The system uses these attempts to refine the grasping model andeventually the system learns to grasp a variety of objects consistently.

In embodiments, learned models for grasping are shared amongst numerousrobots potentially with a wide geographic distribution.

Smart Robotic Kitchen

As discussed herein, the modular robotic kitchen system includes modularcarts, appliances, and transports operable to interact and communicatewith one another to deliver and prepare food according to an optimalschedule and with limited waste.

With reference to FIG. 20, a mobile robot 910 is shown moving a supplycart 920 of food containers to robotic modular unit 930 in accordancewith an embodiment of the invention. In this manner, food supplies canbe provided automatically and without human interaction. Further, therobotic arm of module 930 is operable to pick up and distribute thesupplies as determined by the scheduling engine discussed herein.

Food Quantity Sensors

With reference to FIG. 21, a hot case 950 having sensors 952, 954 isshown. The sensors are mounted such that the contents of the hot casemay be observed to estimate the available quantity of food remaining inaccordance with an embodiment of the invention. In embodiments, thesensor module is used to approximate the amount of food remaining in ahot case. This module provides data about remaining food in a hot caseto a central computer or computer for computing scheduling foodpreparation steps. Also, by ‘hot case’, it is meant a food station thatcontains unpackaged food that can be accessed by customers directly fromthe hot case.

The configuration of the hot case may vary. The hot case 950 shown inFIG. 21 includes a plurality of separate spaces to receive separatecontainers 956. In embodiments, the spaces are operable to warm (ormaintain the temperature) the containers placed therein. However, it isto be understood that the subject invention may include station forcooling food. Indeed, a station may present for pickup or access food,whether temperature controlled or not, to the customers. In embodiments,a sensor module comprises one or more sensor from RGB cameras, IRcameras, depth sensors, or any other imaging sensor.

Additionally, in embodiments, the contents of a hot case is shared withother participants in the robotic kitchen (and sometimes also with amain controller or computer) upon which scheduling decisions (e.g.,scheduling the food preparation steps) are determined.

FIG. 22 is a flow chart for a method 1000 detailing data flow and inputs1010 in a system that drives a demand model 1030 is then used with ascheduler 1040 to control the actions of various Robotic KitchenAssistants 1070, 1072, 1074 in a Smart Robotic Kitchen 1000 inaccordance with an embodiment of the invention.

Step 1030 states demand model. Inputs 1010 to the demand model shown inFIG. 22 include: ad hoc order requests, historical point of sale (POS)data, real-time POS data, regional and national news, calendar events,line length, and other data sources. Still other data sources may beprovided as inputs 1010 including, e.g., a quantity sensor data. Thequantity sensor can feed the demand model as described above inconnection with FIG. 21. Additionally, in embodiments, historicalquantity measurements from many food quantity sensors can be aggregatedand used to improve the accuracy of demand prediction leading to areduction in food waste.

Step 1040 states schedule optimizer. An exemplary scheduling engine isdescribed in US Patent Publication No. 20180345485, entitled“MULTI-SENSOR ARRAY INCLUDING AN IR CAMERA AS PART OF AN AUTOMATEDKITCHEN ASSISTANT SYSTEM FOR RECOGNIZING AND PREPARING FOOD AND RELATEDMETHODS.” In embodiments, a central controller aggregates data to drivescheduling decisions for the entire Smart Robotic Kitchen.

In embodiments, Just-in-Time production scheduling is implemented usingdata from all participants in the Smart Kitchen and drives mechanicaldevices to produce.

The scheduler then directs or instructs one or more robotic kitchenassistant 1070, 1072, 1074 to perform the various food preparation tasksas described herein.

FIG. 22 also shows state management 1050. State management serves tomonitor the state of all tasks of modular robotic kitchen systemincluding for example, current inventory, current preparation step,current state of all items cooking, predicted demand model, executingtasks for robotic systems, and executing tasks of staff. The statemanagement system can allocate tasks to the staff and automation systemsto optimally achieve a predicted demand model. This optimization isupdated multiple times a minute when the state of the system is updatedas food continues to cook, employees succeed or fail in accomplishingtasks, and new orders get placed. An example of a state managementsystem is described in U.S. patent application Ser. No. 16/490,534,filed Aug. 31, 2019, entitled “ROBOTIC KITCHEN ASSISTANT FOR PREPARINGFOOD ITEMS IN A COMMERCIAL KITCHEN AND RELATED METHODS”, U.S. patentapplication Ser. No. 16/490,775, filed Sep. 3, 2019, entitled “AUGMENTEDREALITY-ENHANCED FOOD PREPARATION SYSTEM AND RELATED METHODS”, and USPatent Publication No. 20180345485, filed Aug. 10, 2018, and entitled“MULTI-SENSOR ARRAY INCLUDING AN IR CAMERA AS PART OF AN AUTOMATEDKITCHEN ASSISTANT SYSTEM FOR RECOGNIZING AND PREPARING FOOD AND RELATEDMETHODS”, each of which is incorporated by reference in its entirety forall purposes.

FIG. 23 is block diagram robotic kitchen system 1100 including aconveyor system 1110 in accordance with an embodiment of the invention.The conveyor 1110 is set up to route objects between from a controlledfood environment (e.g., a walk-in refrigerator, storage, vehicle) andtwo or more modular robotic kitchen unit as described herein. Themodular robotic units shown in FIG. 23 include an unboxing robotickitchen assistant 1120, cooking robotic kitchen assistants 1122, 1126,packaging robotic kitchen assistant 1124, packing robotic kitchenassistant 1128, and a distribution robotic kitchen assistant 1130.However, it is to be understood that the number of modular kitchenassistants may vary and be adjusted to suit the kitchen application. Anadditional modular robotic kitchen cart may be conveniently moved intoposition and the system is programmed to operate with the additionalmodular cart as described herein.

In embodiments, the conveyor belt assembly comprises a belt, anenclosure surrounding the belt. The enclosure acts as a protectiveshield to protect moving parts of the conveyor from the food.Additionally, each food item is prepared on a magnetic tray. Inembodiments, the conveyor belt has a series of magnets on it. Theconveyor is operable to move the magnetic food tray from underneath theprotective barrier through a magnetic force.

In embodiments, the conveyor system can include one or more sensors. Forexample, a sensor module can be arranged on one or more of the carts toobtain image data, or time of flight sensing. The sensor moduleoptionally includes one or more CPUs and GPUs. A processor can beprovided that is operable to run convolutional neural networks andgeometric analysis of 3D data achieved through stereographicreconstruction, time-of-flight sensing, or other methods.

Novel Fry Basket

A robotic-friendly fry basket 800 for improved packing efficiency andsafety, and reduced payload on humans is shown in FIGS. 18-19.

FIG. 18A shows a basket design 800 with computer vision (CV) markermount plate 810 and diamond-shaped gripping feature 820. The markermount facilitates location of the basket in 3D space, and the grippingfeature makes it easier for the robot arm to pick up the fryer basketdespite tolerance stacking errors. Examples of the CV marker andgripping feature are described in U.S. patent application Ser. No.16/534,207, filed Aug. 7, 2019, and entitled “ROBOTIC KITCHEN ASSISTANTINCLUDING UNIVERSAL UTENSIL GRIPPING ASSEMBLY”, incorporated herein byreference in its entirety for all purposes.

FIG. 18B shows a basket design with T-shaped feature 860 on top ofbasket to add more vertices to the object with sharp edges. Inembodiments, the basket is designed with features that add additionalsharp vertices for computer vision to pick up on. In embodiments, afryer basket has additional sheet metal features with sharp gradients toenable easy detection and localization via machine learning andtraditional classifier algorithms (see Viola-Jones-type classifiers andAlexNet for examples of what is detectable algorithmically via computervisionhttps://www.cs.cmu.edui-efros/courses/LBMV07/Papers/viola-cvpr-01.pdf).

FIG. 19 shows a basket design with implement 870 for easy dumping ofbasket without lifting full weight of basket. The basket 850 is shownhaving an inverted hook 852 on the front to engage with a horizontal bar870 on a workspace as a pivot. This enables the basket to be dumpedwithout lifting the entire weight of the basket. In embodiments, thebasket has a handle 820 for a robot and a separate handle 854 for ahuman. Consequently, the basket handle is designed in a way that both ahuman and a robot can grab the same handle.

Some of the advantages of the basket described above includes enabling amethod for containing food for cooking in a fryer while enablingcomputer vision localization of basket; reducing time required to cleanafter use; and protecting the human worker. Additionally, inembodiments, smaller baskets are provided and used with the modularrobotic system. Maintaining packing efficiency in a fryer whiledecreasing payload requirements can be accomplished by using manysmaller baskets.

Alternative Embodiments

It is to be understood that the modular robotic kitchen system may varywidely except as recited in the appended claims. For example, in anotherembodiment, and with reference to FIGS. 24A, 24B, a modular robotic cart1200 is shown including a drawer 1210 which grants access to the roboticarm within the shielded workspace 1220, and which limits access to thehuman worker. In contrast, when the drawer is in the open configuration1210′, the robotic arm is prohibited from accessing the drawer, and thehuman worker has access to add or remove contents. In a sense, the noveldrawer design provides a safe human-robot interaction interface tosupply and withdraw food items to the robotic modular cart.

In embodiments, the modular cart may contain a tool belt to hold avariety of tools including measuring tools, gripping tools, andcalibration tools.

In embodiments, the modular cart may have several fixed fiducial markersto provide constant feedback on calibration accuracy and allowinstantaneous calibration.

In embodiments, and with reference to FIG. 25, one or more of themodular robotic arm modules described above may be replaced (orsupplemented) with a frame-mounted linear guide system.

In the embodiment shown in FIG. 25, a robotic arm 1320 is shown coupledto an upper guide rail 1330 via movable base carriage 1340. The upperguide rail is mounted to a portable frame 1350. In the embodiment shownin FIG. 25, the frame 1350 comprises support legs 1352, 1354 and atleast one cross beam 1360 upon which the upper guide rail is fastened.

Feet 1370 are shown extending from the legs at right angles from thelegs for stability. Optionally, the feet may be mounted to the floor.

The carriage and guide cooperate together to axially move the roboticarm along the guide when commanded to do so by the computer processor,which may be located locally, as described above.

Although the linear guide system shows one robotic arm, the invention isnot so limited except where recited in the appended claims. The linearrail guide system may include additional robotic arms movable along therail to further increase the effective reach of the robotic arms. Thecomputer and sensors operate together to determine the food preparationsteps, recognize and locate the food items and utensils, and to scheduleand carry out the order efficiently.

Additionally, the linear guide system may be oriented variously. Inembodiments, a linear guide system extends from the front towards theback (or from the top to bottom) of the cooking area. In addition tosuch axial motion, the robot manipulator itself enjoys several otherdegrees of motion (multi-axis). Consequently, the linear guide systemscan perform any of the skills and applications described above such asthose identified in FIG. 5.

The linear movement may be generated using a number of different linearmovement systems. In embodiments, a cleanable linear actuator designextends the reach of one or more manipulators. In one embodiment, thelinear actuator is composed of a ball screw mechanism with thread andpitch size large enough to easily clean between the threads.

The frame may be made of various materials. In embodiments, the frame isformed of steel tubing, welded together.

Additionally, the linear actuator may be covered to protect it. Inembodiments, a barrier is shaped to cover the sliding mechanisms fromany splashes from food production. A cover allows access of the carriageto move freely along the rail.

Still other techniques may be employed by the robotic kitchen assistantto automatically remove debris from the fryer including rapidlycontacting the rim of a trash receptacle with the skimmer, or brushingthe skimmer with a tool.

Automated Bins

FIG. 26A shows another robotic kitchen system 1400 for preparing food ina restaurant environment in accordance with an embodiment of theinvention. The robotic kitchen system 1400 is shown having robotic arm1410, a plurality of fryers 1420-1422, automated bin station 1430, frameenclosure 1440, safety glass 1441, hot food station 1450, and foodstorage 1460. One or more programmed processors, not shown, can belocated within the enclosure, elsewhere in the vicinity, or cloud-basedfor operating the robotic arm, processing image data from cameras andsensors, and for controlling cooking equipment and other components suchas the automated bins described herein. Indeed, more or fewer componentsmay be included in the system and the invention is not intended to be solimited except as where recited in the appended claims.

Preferably, the robotic arm 1410 is operable to move laterally along aguide or rail mounted to the top of the frame 1440. In embodiments, thisextends the reach of the robotic arm to the full length of theenclosure.

The size of the enclosed robotic kitchen system may vary. Preferably,the enclosure is small, has a small profile and footprint. In theembodiment shown in FIGS. 26A, 26B, the robotic kitchen system 1400includes three fryers 1420-1422, a robotic arm 1410, hood 1427, andautomatic bin station 1430. An exemplary width (W) for the robotickitchen system 1400 ranges from 80 to 130 inches, and more preferably isbetween 90 and 100 inches. An exemplary depth (D) for the robotickitchen system 1400 ranges from 40-60 inches and more preferably isbetween 40-45 inches. An exemplary height (H) for the robotic kitchensystem 1400 ranges from 75-100 inches and more preferably is between75-85 inches.

FIGS. 27-28 are enlarged perspective views of bin station 1430illustrating bin 1442 in a first position for receiving food and asecond position for dumping the food, respectively. Food can be placedby a worker or otherwise deposited in the bin while the bin is in thefirst position shown in FIG. 27. At least a portion of the bins is shownprotruding through window or opening in safety glass 1441. Consequently,the human worker is not in physical contact with the robot, fryer,basket, and hot oil as the case may be.

With reference to FIG. 28, bin 1442′ is shown in a second actuatedposition and any food placed in the bin is gravity fed into the basket1490. The basket 1490 is manipulated within the enclosure by robotic arm1410 shown in FIG. 26, and as described herein.

FIG. 29 shows an enlarged view of a bin 1434. The bin 1434 is shownhaving two parallel side walls 1444, 1446, and a front wall or lip 1448,collectively defining a cavity or channel for food to be placed and heldwhen the bin is in the first position. In embodiments, the floor of thecavity is flat and makes an angle downward between 10-45 degrees whenthe bin is in the first position 1442. However, as described herein, thebin floor may be curved or incorporate other features to channel fooditems to a target area.

Indeed, it is to be understood that the shape of the bin may vary.

Consequently, when food is placed in the bin, the food tends toaccumulate in the cavity and against the inner surface of lip 1448.

The bin shown in FIG. 29 is removably locked with mounting bracket 1436.A wide range of mechanical interlocking features can removably hold thebin to the bracket including for example, without limitation, peg/hole,guide/ridge, slots, fasteners, etc.

The mounting bracket is coupled to the drive shaft 1552 of the motor1438. For example, it can be attached by a shaft collar with a keyway.However, other means may be used to couple the mounting bracket to thedrive shaft of the motor such as a press fit. When the motor isactivated, the drive shaft rotates, moving the attached mountingbracket. This causes the bin to rotate from the first position to thesecond position or vice versa. The second position may go beyond 90° insome cases to help with certain food types which do not slide as easily.Although a motor and rotating drive shaft are described in connectionwith FIG. 29, the invention is not intended to be so limited. Othertypes of actuators may be employed to move or rotate the bin.

With reference to FIG. 30, a rear view of an automated bin station 1500is shown in accordance with another embodiment of the invention.Automated bin station is shown having three bins 1510,1520,1530 each ofwhich has a dedicated mounting bracket and motor. The bins are slopeddownward toward the front.

A sensor array 1540 is shown mounted on safety shield 1550. As describedherein the sensor array may comprise a camera and supplies image data tothe programmed processor for monitoring the position, contents andmovement of the bins.

FIG. 31 is a flow chart for a method 1700 to receive food, cook andtransfer hot food to a holding area in accordance with an embodiment ofthe invention.

Step 1710 states to load bin. An example of a bin 1434 is shown in FIGS.26-29. The front end of the bins protrude from an opening in the safetyshield 1441. A human or robot may gather food from a storage area suchas cold storage 1460 and dump the food into the bin(s). Preferably, asshown, the bins are arranged for a “direct drop” from the bag into thebin. The bin contains the food within the parallel walls 1444, 1446 andlip 1448 of the bin. The bin is arranged rotated or tilted forward intothe first position such that the food remains in the bin, and does notslide out the open rear end of the bin. A wide range of types of food,whether fresh or frozen, may be added to the bin such as, for example,fries, chips, chicken, vegetables, fish, etc. Desirably, the entireloading step 1710 is performed outside of the enclosure defined by frame1440 and shield 1441, thereby physically separating the worker from themoving robotic arm, cooking equipment, and potentially dangerous hotequipment and materials such as hot baskets, hot cooking oil.

Additionally, in embodiments, the bins are removable from the mountingbrackets for convenient food scooping and cleaning. The bins may alsovary in size and shape. The bins may be sized to accommodate arestaurant's menu or needs.

Bins may also be color coded. Color-coded bins add to safety byassisting food prep workers to avoid food cross-contamination (i.e.,shellfish or dairy-based can remain separate from other foods). Inembodiments, the bins are configured to be stackable with one anotherand can be stored in a stacked arrangement 1424 as shown in FIG. 26.

Step 1720 states to classify food and determine cook schedule. Asdescribed herein, the food is classified using a programmed processorand based on image data from cameras aimed at the food bins. Theprocessor further is operable to determine an optimal schedule to cookthe food in the bin in view of the current state of the system.

Step 1730 states to position basket for catching food from bin. Withreference to FIG. 27, the robotic arm manipulates the basket 1490beneath the rear end of the bin holding the food.

Step 1740 states to automatically dump the food to the basket. Withreference to FIG. 28, the bin is moved (preferably rotated) to a second“dumping” position 1442′, causing the food within the bin to sliderearwardly into the basket 1490.

Step 1750 states to move basket to fryer and cook food. With referenceto 26A, robotic arm 1410 is operable to move basket 1490 into one of thefryers 1420-1422 and the automated bins are returned to the firstposition shown in FIG. 27.

The processor is operable to determine the cooking time, duration, andoptionally, measure the doneness as discussed herein. The system isoperable to cook multiple foods in parallel arising from the multipleautomated bins and fryers. Although three automated bins are shown, moreor less automated bins may be incorporated into the design. For example,a modular automated bin station may be positioned on each side of thefryers and within the safety shield.

Step 1760 states to move basket and dump food to hot holding. Withreference to FIGS. 32A, 32B, and after the food is cooked in the fryer1420, the robotic arm 1410 moves basket 1490 to slide, chute, or ramp1480. The timing and position of the robotic arm is controlled by theprocessor and state of the system based on recipe information and cameraimage data. The robotic arm, while within the enclosure, dumps thecooked food onto ramp 1480. The food falls/slides downward, through anopening 1482 in the safety shield, and into hot holding area 1450 wherethe cooked food may be sorted and bagged.

FIG. 33 shows a food item sorting assembly 2000 in accordance withanother embodiment of the invention. The food item sorting assembly 2000is shown having a food bin 2010 positioned underneath an outbound edgeof chute 2020. A linear rail 2040 and tilt motor 2050 are operable tomove the bin along the rail. A computer (e.g., the computer describedabove for controlling the robotic arm) is programmed and operable todirect the bin to a desired food holding tray 2060, where the bin can berotated to dump the food items into the desired location.

Each food holding tray 2060 includes a ramp (or chute) portion 2064which guides or channels the food dumped thereon towards the lower baseportion 2062. The human workers (not shown) may access the food items inthe base portion of the food stations 2060.

Although four food holding trays 2060 are shown in FIG. 33, theinvention is not intended to be so limited. More or fewer food holdingtrays may be incorporated into the sorting system.

Optionally, a heat lamp 2070 is arranged above one or more of the foodholding trays to keep the food warm.

Optionally, a safety shield 2080 is arranged between the food bin 2010and the human workers. The shield 2080 is shaped to cover bin, rail andmotor yet permit food items to fall into the base of the food tray. Morespecifically, in the embodiment shown in FIG. 33, the shield 2080 is (a)vertically arranged above the base 2062 of the food holding tray and (b)set back from the proximal edge of the tray. Consequently, the humanworkers can access to the food items dumped into the tray yet areprohibited from contacting the moving components of the automatedsorting system (e.g., the motor, bin, etc.).

FIGS. 34a-34c illustrate sequential steps in a method for sorting fooditems in accordance with one embodiment of the invention. With referenceto FIG. 34a , a food bin 2010 is shown positioned at a slight angle andbelow the chute 2020 for receiving food items from the chute. The chute2020 protrudes through a window of the robotic arm enclosure.

It is to be understood that the food item sorting systems describedherein may also be operable with and/or receive food items from a widevariety of sources not limited to robotic systems. Sources may include,without limitation, human workers, automated equipment, or other systemswhether robotic or non-robotic based.

FIG. 34b shows the food bin 2010 transported laterally (L) along thelinear rail to a desired location above a desired food holding tray2070.

FIG. 34c shows rotating (R) the food bin 2010 to dump the food into thedesired food holding tray 2070.

Optionally, the bin may be returned to the position shown in FIG. 34a ,namely, home position.

The sorting system desirably operates automatically to sort a widevariety of foods received from the robotic arm enclosure 2030. Thesorting system includes a computer (preferably the same computer usedfor the robotic arm) to determine how to position and orient the foodbin, and the timing to do so. In embodiments, the sorting system isoperable to place food items into the different food holding trays basedon the type of food.

Optionally, the system stores the location of the food, food identify,time at the location, and food status in memory. Optionally, thecomputer activates an alert after a threshold time or condition is met.For example, if the amount of time the food is located in the foodholding tray exceeds a threshold time for freshness, an alert isactivated.

The type of alert may vary. Examples of alerts include audio sounds anddisplay graphics intended to notify the human worker to remove the foodfrom the food holding tray. The human worker may input the status of theorder or optionally, the sorting system may be provided with sensors orcameras to obtain image data of the food holding tray areas in order forthe computer to compute the status and state of the food in each of thefood holding trays. The state of the system is then updated forinventory in food holding areas.

Optionally, the computer is programmed to maintain inventories in thefood holding trays. First, inventory states are automatically determinedbased on quantity observed, freshness or time in holding areas, foodtypes, and order anticipation (e.g., increase inventory during rushperiods). Preferably a trained model for the food holding trays isincorporated into the system to recognize or estimate the quantity offood present in a food holding tray. Such a model may be a computervision model (e.g., a CNN model) and trained by placing different knownquantities of food in each of the trays and instructing the model as tothe actual quantity present in the tray. Features or inputs to the modelinclude the type of food item, diameter or characteristic length, etc.

Additionally, the computer may be programmed to compute the temperaturein hot food holding areas based on image data or temperature sensors. Analert or notification may be computed and sent to the human worker whenthe temperature exceeds a threshold value. In embodiments, the hot foodholding model is trained to recognize the temperature of the food itemsin the food holding bins based on thermal signatures arising fromthermal cameras or sensors.

Additionally, in embodiments, data is perpetually collected from themultiple sorting and automated bin stations and sent to a remote server(e.g., a cloud based server) for aggregating and/or training acomprehensive model to recognize ingredients placed in the bins, trackfood holding times, compute temperatures, etc.

The computer then evaluates whether to instruct or notify the worker tocarry out a task. Examples of tasks include, without limitation (a) toadd or supply more food to the tray based on the inventory levels or(b)to throw away the food in the food holding areas if the food isdetermined to be expired.

Advantages of embodiments of the robotic kitchen system with automatedbins and sorting, and related methods, include without limitation: (a)robot-free safety zone, by use of an entirely enclosed system serving toprotect humans from hazardous equipment and materials (no human-robotworking zones); (b) contamination control, by use of color-coded bins tofacilitate adherence to standard operating procedures, specificallythose designed to prevent cross-contamination; (c) optimizes human labortime by automating certain steps in the cooking, namely, frying process;(d) robustness, namely, tireless low maintenance robotic action; (e)flexibility, namely, the system accepts wide range of types of foodsregardless of shapes, size, and quantity; (f) safety, by physicallyseparates human from moving robotic equipment and minimizing oreliminating need of safety sensors; (g) manually lifting baskets iseliminated as well as slip and falls in vicinity of fryers; (h) lessfood contact with human and equipment reduces chances of contamination;(i) automatic sorting to storage trays, and (j) modularity, the systemis modular and expandable.

1. A robotic kitchen system for preparing food items in combination withat least one kitchen appliance in a commercial or restaurant kitchen,the robotic kitchen system comprising: a robotic arm and end effector; abasket, held by the end effector; an automated bin assembly in thevicinity of the robotic arm, said automated bin assembly comprising abin for holding the food items; a camera or sensor array for collectingimage data of the food items in the bin; and a central processoroperable to compute and provide directions to the robotic arm andautomated bin assembly to (a) move the robotic arm to the bin; and (b)actuate the bin to drop the food items from the bin into the basket. 2.The robotic kitchen system of claim 1, wherein the central processor isfurther operable to compute and provide directions to move the basketinto a fryer after the basket receives the food items.
 3. The robotickitchen system of claim 2, further comprising a shield and first windowin the shield through which at least a portion of the bin can protrudeto receive the food items from outside the shield.
 4. The robotickitchen system of claim 3, wherein the bin is operable to rotate from afirst position in which a front portion of the bin protrudes through thefirst window, and a second position in which the food items fall out arear opening of the bin.
 5. The robotic kitchen system of claim 3,wherein the central processor is further operable to compute and providedirections to transfer the food items through a second window after thefood items are cooked in the fryer.
 6. The robotic kitchen system ofclaim 1, further comprising: a ramp extending through the second window,and arranged at a downward angle such that the food items placed on theramp from within the enclosure slide down the ramp out the secondwindow; a sorting bin for collecting the food from the ramp; a linearrail and motor for moving the sorting bin laterally and rotationally;and at least one food holding tray for dumping the cooked food therein,and wherein the central processor is further operable to compute andprovide directions to move the sorting bin to the at least one foodholding tray, and to dump the food into the food holding tray.
 7. Therobotic kitchen system of claim 1, wherein the automated bin stationcomprises a plurality of bins automatically rotatable.
 8. The robotickitchen system of claim 7, comprising a plurality of fryers.
 9. Therobotic kitchen system of claim 8, further comprising a schedulingengine to determine a sequence of food preparation steps to detect andtransfer the food items from each of the bins to the fryers for cookingbased on a plurality of inputs selected from the group consisting ofcamera data, customer orders, inventory, and recipe information.
 10. Therobotic kitchen system of claim 3, further comprising a frame, to whichthe shield is attached, and a linear guide to which the robotic arm ismovable coupled.
 11. An automated food bin station for transferring fooditems from a human to robotic kitchen assistant comprises: a bin; amotor removably coupled to the bin and operable to move the bin betweena first position for receiving the food items and to a second positionfor dumping the food items; a sensor array for obtaining image data ofthe bin; a processor for detecting and classifying food items in thebin, and for instructing the motor to move the bin between the firstposition and the second position.
 12. The automated food bin station ofclaim 11, further comprising a bracket for coupling each bin to eachmotor.
 13. The automated food bin station of claim 11, furthercomprising a frame for supporting a first shelf upon which the motor andbin are mounted.
 14. The automated food bin station of claim 13, furthercomprising a storage shelf, and a plurality of bins in a stackedarrangement on the storage shelf.
 15. The automated food bin station ofclaim 14, wherein the plurality of bins comprises a first bin type and asecond shaped bin type larger than the first bin type.
 16. The automatedfood bin station of claim 13, further comprising a catch tray arrangedbelow the bin for catching food items.
 17. The automated food binstation of claim 11, further comprising a safety shield defining anopening through which a portion of the bin protrudes.
 18. The automatedfood bin station of claim 11, further comprising a plurality of bins,each bin of said bins is coupled to an individual motor such that eachbin of said bins is independently movable to catch or dump the fooditems.
 19. The automated food bin station of claim 11, wherein the motorprovides rotational motion.